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Program description 



Check & Connect is a dropout prevention strategy that reiies 
on ciose monitoring of schooi performance, as weii as mentor- 
ing, case management, and other supports. The program has 
two main components: “Check” and “Connect.” The Check 
component is designed to continually assess student engage- 
ment through close monitoring of student performance and 
progress indicators. The Connect component involves program 
staff giving individualized attention to students, in partnership 



with schooi personnel, family members, and community service 
providers. Students enrolled in Check & Connect are assigned a 
“monitor” who reguiarly reviews their performance (in particuiar, 
whether students are having attendance, behavior, or academic 
probiems) and intervenes when problems are identified. The 
monitor also advocates for students, coordinates services, pro- 
vides ongoing feedback and encouragement, and emphasizes 
the importance of staying in schooi. 



Research 



One study of Check & Connect met the What Works Ciearing- 
house (WWC) evidence standards, and a second study met WWC 
standards with reservations. The two studies included a total of 
more than 200 students attending Minneapolis high schools. In 



both studies the students entered the program at the beginning 
of the ninth grade. The studies examined the program’s effects in 
three dropout prevention domains considered by the WWC: stay- 
ing in school, progressing in school, and completing school.^ 



Effectiveness 



Check & Connect was found to have positive effects on staying in school and potentially positive effects on progressing in school. It 
was found to have no discernible effects on completing school within four years of entering the program. 



Rating of effectiveness 
Improvement index^ 



Staying in schooi 

Positive effects 

Average: +25 percentile points 
Range: +18 to +31 percentile points 



Progressing in schooi 

Potentially positive effects 
Average: +30 percentile points 
Range: +30 percentile points 



Compieting schooi 

No discernible effects 
Average: +1 percentile point 
Range: +1 percentile point 



1 . To date, there are only a few studies of the effectiveness of Check & Connect. Findings and conclusions may change as new research becomes available. 

2. These numbers show the average and range of improvement indices for findings in the two studies within the three domains. 
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Developer and contact 

Check & Connect was developed by the Institute on Community 
Integration at the University of Minnesota, as a partnership of 
researchers, practitioners, parents, and students. More informa- 
tion and additional references to research about the program 
can be found at http://ici.umn.edu/checkandconnect . or by 
sending an email to info@icimail.education.umn.edu. 

Scope of use 

The Check & Connect model has been used in Minneapoiis 
pubiic schools with middie and high school students who have 
learning, emotionai, and behaviorai disabiiities. The model 
has been replicated in eight school districts in Dakota County, 
Minnesota as part of an initiative targeted at chronically truant 
youth. The model also has been used in three school districts in 
Dakota County in an effort to reduce truancy among elementary 
school students. 

Description of intervention 

Check & Connect has two main components: “Check” and 
“Connect.” The Check component is designed to continually 
assess student engagement through close monitoring of student 
performance and progress indicators (including the student’s 
attendance, incidence of suspensions, course grades, and 
credits). The Connect component involves program staff giving 
individualized attention to students, in partnership with school 
personnel, family members, and community service providers. 

These program components are implemented by the Check 
& Connect “monitor,” who functions as the student’s mentor and 



case worker. Monitors provide basic intervention for all students 
on their caseload, as well as intensive intervention — which is 
more frequent and individualized — for students as needed. Basic 
interventions involve regular structured discussions between 
the monitor and student — at least twice a month for second- 
ary students and weekly for elementary and middle school 
students — about their progress in school and problem-solving 
steps to resolve conflict and cope with challenges. When 
intensive interventions are required because of particularly poor 
attendance or school performance, they are tailored to students’ 
specific circumstances. Intensive interventions focus on three 
areas of support: problem-solving (including mediation and 
social skills development), academic support (through homework 
assistance, schedule changes, and tutoring), and recreational 
and community service exploration. In addition, the program 
focuses on family outreach, with frequent contact and collabora- 
tion between home and school. 

In the Check & Connect programs described in this interven- 
tion report, monitoring positions were staffed by graduate stu- 
dents and community members with either bachelors degrees in 
human services-related fields or equivalent experience. Program 
coordinators, who oversaw the program and supervised the 
monitors, were typically special education coordinators, school 
psychologists, or special education resource teachers. 

Cost 

Program developers report that implementing Check & Connect 
in secondary schools cost about $1,400 a student per year in the 
2001-02 school year.^ 



Six studies reviewed by the WWC investigated the effects of the 
Check & Connect program. One study met WWC evidence stan- 
dards, and a second study met evidence standards with reser- 
vations. Two studies did not meet WWC relevance screens — one 



did not focus on the relevant student age range (middle and 
high school) and the other did not examine outcomes from the 
three domains relevant for the review. The two remaining studies 



3. See Sinclair, M., & Kaibel, C. (2002). Dakota County: School success Check and Connect program evaluation, 2002 final summary report. Minneapolis, 
MN: University of Minnesota, Institute on Community Integration. 
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did not meet WWC evidence screens because they iacked an 
equivalent comparison group. 

The study that met WWC evidence standards (Sinciair, 
Christenson, Eveio, & Huriey, 1998) was a randomized controiied 
trial that included 94 high school students from the Minneapolis 
public schools with learning, emotional, or behavioral disabilities. 
Students were randomly assigned at the beginning of ninth 
grade, with 47 students assigned to the treatment group and 
47 students assigned to the control group. In this study, both 
treatment and control group students received Check & Connect 
services in seventh and eighth grade, but only treatment group 
students continued to receive these services in ninth grade. 



The study that met evidence standards with reservations 
(Sinclair, Christenson, SThurlow, 2005) was a randomized 
controlled trial with a relatively large attrition rate — slightly 
more than 30% of those originally assigned."' The post-attrition 
sample included 144 ninth-grade students from Minneapolis 
public schools with emotional or behavioral disabilities, includ- 
ing 71 students randomly assigned to the treatment group and 
73 students randomly assigned to the control group. In this 
study, treatment group students received Check & Connect 
services throughout high school, while the control group 
received no Check & Connect services. 



Findings 

The WWC review of dropout prevention programs addresses 
student outcomes in three domains: staying in school, progress- 
ing in school, and completing school. 

Staying in schooi. The Sinclair and colleagues (1998) study 
reported that ninth grade students enrolled in Check & Connect 
were significantly less likely than similar control group students 
to have dropped out of school at the end of the first follow-up 
year (corresponding to the end of the freshman year) — 9% 
compared with 30%. The Sinclair, Christenson, &Thurlow (2005) 
study reported that Check & Connect students were significantly 
less likely to have dropped out of school at the end of the fourth 
follow-up year (corresponding to the senior year for students 
making normal progress) — 39% compared with 58%. 

Progressing in schooi. The Sinclair and colleagues (1998) 
study reported that students in Check & Connect earned 
significantly more credits toward high school completion during 
ninth grade than did students in the control group. The Sinclair, 
Christenson, SThurlow (2005) study did not report on high 
school credit outcomes. 



Compieting schooi. The Sinclair, Christenson, &Thurlow 
(2005) study examined Check & Connect’s effect on whether 
students completed school “on time” (within four years of 
entering the ninth grade). The study indicated that there was 
no statistically significant or substantially important effect on 
on-time high school completion. At the end of the four-year 
follow-up period, combining receipt of high school diplomas and 
GED certificates, rates of on-time completion were about the 
same for Check & Connect and control group students — 30% 
compared with 29%. (At this point, 31% of intervention students 
and 14% of control students were still enrolled in school but had 
not yet graduated.) Because of its short follow-up period, the 
Sinclair and colleagues (1998) study did not examine impacts on 
school completion. 

Rating of effectiveness 

The WWC rates interventions as positive, potentially positive, 
mixed, no discernible effects, potentially negative, or negative. 
The rating of effectiveness takes into account four factors: the 
quality of the research design, the statistical significance of the 



4. In this study, 206 students were randomly assigned, with random assignment occurring prior to receiving parental permission. Of those originally 
assigned, 26 refused to participate either before or after signing permission forms. An additional 36 students were dropped from the sample because, 
during the first year of the study, they moved out of district, entered a correctional institution, or could not be located. This represents a total loss of 
sample of 62 students — 30.1 % of those originally assigned. 
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The Vim touna Check & 
Connect to have positive 
effects on staying in schooi, 
potentiaiiy positive effects 
on progressing in schooi, 
and no discernibie effects 
on schooi compietion 



References 



findings, the size of the difference between participants in the 
intervention condition and comparison condition, and the con- 
sistency in findings across studies (see the WWC Intervention 
Rating Scheme) . 

Improvement index 

For each outcome domain, the WWC computed an improvement 
index based on the average effect size (see the Technical Details 
of WWC Conducted Computations) . This improvement index rep- 
resents the difference between the percentiie rank of the average 
student in the intervention condition versus the percentile rank of 
the average student in the comparison condition. Uniike the rating 
of effectiveness, the improvement index is entirely based on the 
size of the effect, regardless of the statistical significance of the 
effect, the study design, or the type of analysis. The improvement 
index can take on values between -50 and +50, with positive 
numbers denoting favorable results. The average improvement 
index for staying in school is +25 percentile points, with a range of 
+18 to +31 percentile points across the two studies. The improve- 



ment index for progressing in school is +30 percentile points. The 
improvement index for completing school is +1 percentile point. 

Summary 

The WWC reviewed six studies on Check & Connect that were 
designed to assess the program’s effectiveness. Four of these 
studies passed WWC relevance screens — they focused on the 
program’s effectiveness among middle and high school students 
and examined outcomes from at least one of the three relevant 
domains: staying in school, progressing in school, and complet- 
ing school. Of these four studies, one met WWC evidence 
standards and another met WWC evidence standards with res- 
ervations. These two studies found positive effects on staying in 
school and potentially positive effects on progressing in school. 
The studies found no discernible effects on completing school 
on time (within four years of entering ninth grade). The conclu- 
sions presented in this report may change as new research on 
Check & Connect emerges. 



Met WWC evidence standards 

Sinclair, M. F., Christenson, S. L., Evelo, D. L., & Hurley, C. M. 
(1998). Dropout prevention for youth with disabilities: Efficacy 
of a sustained school engagement procedure. Exceptional 
Children, 65(1), 7-21. 

Additional sources: 

Christenson, S. L., Sinclair, M. F., Thurlow, M. L., & Evelo, D. 
(1999). Promoting student engagement with school using 
the Check & Connect model. Australian Journal of Guid- 
ance & Counseling, 9(1), 169-184. 

Sinclair, M. F., Christenson, S. L., Lehr, C. A., & Anderson, 

A. R. (2003). Facilitating student engagement: Lessons 
learned from Check & Connect longitudinal studies. The 
California School Psychologist, 8(1), 29-42. 



Met WWC evidence standards with reservations 

Sinclair, M. F., Christenson, S. L., & Thurlow, M. L. (2005). Promot- 
ing school completion of urban secondary youth with emotional 
or behavioral disabilities. Exceptional Children, 77(4), 465-482. 

Additional sources: 

Sinclair, M. F., Christenson, S. L., Evelo, D. L., Hurley, C. M., 
Kau, M. Y., Logan, D. T, Thurlow, M. L., & Westberry, D. 
(2001). Persistence Plus: Using Check & Connect proce- 
dures to improve service delivery and positive post-school 
outcomes for secondary students with serious emotional 
disturbance. (CDFA No. 84.237H). Minneapolis, MN: Uni- 
versity of Minnesota, Institute on Community Integration. 
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R6f6r6nC6S (continued) Did not meet WWC relevance screens 



Sinclair, M. F. & Lehr, C. A. (2000). Dakota County: Elemen- 
tary Check & Connect programs. Annual summative 
program evaluation report. Minneapolis, MN: University of 
Minnesota, Institute on Community Integration. 



Anderson, A. R., Christenson, S. L., Sinclair, M. F., & Lehr, C. A. 
(2004). Check & Connect: The importance of relationships 
for promoting engagement with school. Journal of School 
Psychology, 42, 95-113.® 



Lehr, C. A., Sinclair, M. R, & Christenson, S. L. (2004). Address- 
ing student engagement and truancy prevention during the 
elementary school years: A replication study of the Check 
& Connect model. Journal of Education for Students Placed 
At-Risk, 9(3), 279-301.® 

Additional sources: 

Sinclair, M. F. & Lehr, C. A. (2001). Dakota County: Elementary 



Thorton, FI. E. (Ed.). (1995). Staying in school: A technical 



Sinclair, M. F. & Kaibel, C. (2002). Dakota County: Secondary 



Did not meet WWC evidence screens 



Check & Connect programs. Program evaluation 2002 final 
summary report. Minneapolis, MN: University of Minnesota, 
Institute on Community Integration.^ 



report of three dropout prevention projects for middle school 
students with learning and emotional disabilities. Technical 
report 1990-1995. ABC dropout prevention and intervention 
series. Washington, DC: U.S. Department of Education.® 



Check & Connect programs. Program evaluation 2001 
summary report. Minneapolis, MN: University of Minne- 
sota, Institute on Community Integration. 



For more information about specific studies and WWC caicuiations, piease see the WWC Check & Connect 
Technicai Appendices . 



5. The outcome measures are not relevant to this review. 

6. The sample is not appropriate this review: the study did not include middle school or high school students. 

7. Does not use a strong causal design: the study did not use a comparison group. 

8. Does not use a strong causal design: the study used a nonequivalent comparison group. 
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Appendix 



Appendix A1.1 


Study characteristics: Sinciair, Christenson, Eveio, & Huriey, 1998 (randomized controiied triai) 


Characteristic 


Description 


Study citation 


Sinclair, M. R, Christenson, S. L., Eveio, D. L., & Huriey, C. M. (1998). Dropout prevention for youth with disabiiities: Effioacy of a sustained school engagement procedure. 
Exceptional Children, 65(1), 7-21. 


Study design 


Study used a random assignment research design and included 94 high school students — 47 in the intervention group and 47 in the control group. Students were randomly 
assigned just before entering the ninth grade. 


Participants 


Partioipants were special education students enrolled in ninth grade during the 1994-95 school year who were classified with a learning, emotional, or behavioral disability. 
Learning disabilities were the most common classification, with 75% of participants having this diagnosis. A little more than 40% of participants were classified as having a 
severe disability. Most participants were African-American (59%); most were males (68%); and most participated in the free or reduced-price lunch program (71 %). Students 
were 15-years-old, on average, when they entered ninth grade. 


Setting 


Study was conducted in Minneapolis public high schools. 


Intervention 


The intervention group received Check & Connect services in the seventh and eighth grade and, after being assigned to the intervention group, continued to receive the 
program in ninth grade. Students in Check & Connects their attendance, behavior, and academic performance observed on a daily basis by their “monitor,” who also 
functioned as a mentor and caseworker. Monitors met with students at least twice a month and more often when acute attendance, performance, or behavior problems arose. 


Control 


Control group students received Check & Connect m seventh and eighth grade but, after assignment to the control group, did not continue to receive these services when they 
entered high school. Control group students attended the same set of high schools attended by intervention group students. 


Primary outcomes 
and measurement 


Two relevant outcomes from this study are included in this summary: the percentage of students who had dropped out at the end of the first follow-up year and the number of 
credits earned during the first follow-up year. (See Appendix A3.) 


Staff training 


No specific information concerning staff training was provided, however, program staff noted that monitors should possess several key attributes. In particular, they indicated 
that “(w)hile familiarity with the schools and community resources was desirable, the essential qualifications of a monitor included patience; a belief that all students have 
abilities; willingness to work cooperatively with families and staff; and advocacy skills, particularly communication skills, such as the ability to negotiate, compromise, and 
confront conflict constructively” (p.10). 
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Appendix A1.2 Study characteristics: Sinciair, Christenson, & Thuriow, 2005 (randomized controiied triai) 



Characteristic 


Description 


Study citation 


Sinclair, M. R, Christenson, S. L., & Thuriow, M. L. (2005). Promoting school completion of urban secondary youth with emotional or behavioral disabilities. Exceptional 
Children, 7?(4), 465-482. 


Study design 


The study used a random assignment research design. The post-attrition sample included 144 high school students — 71 in the intervention group and 73 in the control 
group. Students were randomly assigned at the beginning of ninth grade. 


Participants 


This replication of Check & Connect included special education students who entered ninth grade in 1996 and 1997. To be eligible for the study, participants had to be classi- 
fied as having an emotional or behavioral disorder. Most students were African-American (64%); most were males (84%); and most participated in the tree or reduced-price 
lunch program (70%). Students were 14.5-years-old, on average, when they entered ninth grade. 


Setting 


Study was conducted in Minneapolis public high schools. 


Intervention 


The intervention group participated in Check & Cormecfforfouryears, starting in ninth grade. Students had their attendance, behavior, and academic performance observed 
on a daily basis by their “monitor,” who also functioned as a mentor and case worker. The monitor stayed with the student even it the student transferred to another school 
within the district. Monitors met with students at least twice a month and more often when acute attendance, performance, or behavior problems arose. 


Control 


Control group students attended the same schools as intervention students but did not receive Check & Connect. 


Primary outcomes 
and measurement 


Relevant outcomes included in this study are: the percentage of students who had dropped out of school at the end of the fourth year following random assignment and the 
percentage of students who either completed high school or their GED by the end of the fourth year. (See Appendix A3.) 


Staff training 


Monitors participated in an initial orientation workshop. They also attended weekly or biweekly staff meetings and periodic statf development sessions. Each monitor received 
instructions on how to complete the monitoring sheet to ensure consistency across monitors and settings. Monitors submitted printouts ot attendance records with their 
monitoring sheets for verification purposes. 
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Appendix A2.1 Outcome measures for the staying in schooi domain 



Outcome measure 


Description 


Dropped out of school 
at end of first year after 
random assignment 


This measure represents whether students were not enrolled in school at the end of the first academic year after random assignment — or the end of the ninth grade, 
because students were randomly assigned at the beginning of high school. School enrollment was verified through a tracking system established for the study and was 
drawn from various sources, including project and school staff, social workers, and probation offioers, as well as the school district’s on-line database (as cited in Sinclair 
et al„ 1998), 


Dropped out of school at 
end of fourth year after 
random assignment 


This measure represents students who had not completed high school or a GED and were not enrolled in school at the end of the fourth academic year after random 
assignment — or the senior year for those making normal progress toward graduation, because students were randomly assigned at the beginning of ninth grade. The study 
authors counted students as enrolled (and thus having not dropped out) if they transferred to another school district, a nonpublic school, or a state-approved education 
program, or if they were in a correctional institution (as cited in Sinclair, Christenson, & Thurlow, 2005). 



Appendix A2.2 Outcome measures for the progressing in schooi domain 



Outcome measure 


Description 


Credits earned during first year 
after random assignment 


This measure was drawn from the school district’s on-line database and represents the total credits earned during the first academic year after random assignment — or 
the ninth-grade school year, because students were randomly assigned at the beginning of high school (as cited in Sinclair et al., 1998). 



Appendix A2.3 Outcome measures for the compieting schooi domain 



Outcome measure 


Description 


Completed high school diploma 
or GED by end of fourth year 
after random assignment 


This measure represents the percentage of students who completed an education program, including graduation with a standard diploma or a GED certificate, by the end of 
the fourth academic year after random assignment — or the senior year for those making normal progress toward graduation, because students were randomly assigned at 
the beginning of ninth grade. Eligh school diploma receipt was verified through school district records. GED completion was verified independently by the State Department 
of Education (as cited in Sinclair, Christenson, & Thurlow, 2005). 
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Appendix A3.1 Summary of study findings inciuded in the rating for the staying in schooi domain^ 









Author’s findings from the study 














Mean outcome 
(Standard deviation 




WWC calculations 




Outcome measure 


Study 

sample 


Sample size 
(schools/ 
students) 


Check & Connect 

group Control group 


Mean difference^ 
Check & Connect 
- control 


Statistical 
significance^ 
Effect size'' (at a = 0.05) 


Improvement 

index^ 



Sinclair et al., 1998 (randomized controlled trial)^ 



Dropped out of school (%) Grade 9 94 


9 

(28) 


30 

(46) 


-21 


0.88 


Statistically 

significant 


31 


Average® for staying in school (Sinclair et al., 1998) 








0.88 


Statistically 

significant 


31 


Sinclair et al., 2005 (randomized controlled trial with attrition problems)'' 


Dropped out of school (%) Grade 12 144 


39 

(49) 


58 

(50) 


-18 


0.46 


Statistically 

significant 


18 


Average® for staying in school (Sinclair et al., 2005) 








0.46 


Statistically 

significant 


18 


Domain average® for staying in school across all studies 








0.67 


na 


25 



na = not applicable 

1 . This appendix reports findings considered for the effectiveness rating and the average improvement indices. The two Sinciair et ai. studies aiso examined additionai outcomes that do not faii 
within the three domains addressed by the WWC review — staying in school, progressing in school, and completing school. In the Sinclair et al. (1998) study, these additional outcomes included 
attendance, assignment completion, teacher’s perceptions of the student’s academic competence and problem behaviors, and student perceptions of the relevance of school and the likelihood 
of graduation. In the Sinclair et al. (2005) study, these additional outcomes included the likelihood that the student would transfer to another school and the level of the student’s participation in 
the special education transition program. These additional results are not summarized in this report. 

2. The standard deviation across all students in each group shows how dispersed the participants’ outcomes are: a smaller standard deviation on a given measure would indicate that participants 

had more similar outcomes. For binary (“zero-one”) outcomes, standard deviations were calculated using the following formula, ^n/(n -l)* Vp *0 -pj ’ P percentage of the sample 

with a value of 1 for the outcome and N is the sample size. 

3. Positive effect sizes represent effects in the desired direction; negative effect sizes represent effects in the undesired direction. 

4. For an explanation of the effect size calculation, see Technical Details of WWC-Conducted Computations . 

5. Statistical significance is the probability that the difference between groups is a result of chance rather than a real difference between the groups. 

6. The improvement index represents the difference between the percentile rank of the average student in the intervention condition and that of the average student in the comparison condition. 
The improvement index can take on values between -50 and +50, with positive numbers denoting favorable results. 

7. The level of statistical significance was reported by the study authors or, where necessary, calculated by the WWC to correct for clustering within classrooms or schools and for multiple com- 
parisons. For an explanation about the clustering correction, see the WWC Tutorial on Mismatch . See Technical Details of WWC-Conducted Computations for the formulas the WWC used to 
calculate statistical significance. In the case of both Sinclair et al. studies, no corrections for clustering or multiple comparisons were needed. 

8. The WWC-computed average effect sizes for each study and for the domain across studies are simple averages rounded to two decimal places. The average improvement indices are calculated 
from the average effect size. 
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Appendix A3.2 Summary of study findings inciuded in the rating for the progressing in schooi domain^ 









Author’s findings from the study 














Mean outcome 
(Standard deviation ^) 




WWC calculations 




Outcome measure 


Study 

sample 


Sample size 
(schools/ 
students) 


Check & Connect 

group Control group 


Mean difference® 
Check & Connect 
- control 


Statistical 
significance® 
Effect size^ (at a = 0.05) 


Improvement 

index® 



Credits earned Grade 9 

Domain average® for progressing in school 



Sinclair et ai., 1998 (randomized controlled trial)^ 

92 12.13 6.63 5.50 



12.13 

(6.56) 



6.63 

(6.63) 



0.83 Statistically 30 

significant 

0.83 Statistically 30 

significant 



1. This appendix reports findings considered for the effectiveness rating and the average improvement index. Sinclair et al. (1998) also examined additional outcomes that do not fall within the 
three domains addressed by the WWC review — staying in school, progressing in school, and completing school. These additional outcomes included attendance, assignment completion, 
teacher’s perceptions of the student’s academic competence and problem behaviors, and student perceptions of the relevance of school and the likelihood of graduation. These additional 
results are not summarized in this report. 

2. The standard deviation across all students in each group shows how dispersed the participants’ outcomes are: a smaller standard deviation on a given measure would indicate that participants 
had more similar outcomes. 

3. Positive effect sizes represent effects in the desired direction; negative effect sizes represent effects in the undesired direction. 

4. For an explanation of the effect size calculation, see Technical Details of WWC-Conducted Computations . 

5. Statistical significance is the probability that the difference between groups is a result of chance rather than a real difference between the groups. 

6. The improvement index represents the difference between the percentile rank of the average student in the intervention condition and that of the average student in the comparison condition. 
The improvement index can take on values between -50 and +50, with positive numbers denoting favorable results. 

7. The level of statistical significance was reported by the study authors or, where necessary, calculated by the WWC to correct for clustering within classrooms or schools and for multiple 
comparisons. For an explanation about the clustering correction, see the WWC Tutorial on Mismatch . See Technical Details of WWC-Conducted Computations for the formulas the WWC used to 
calculate statistical significance. In the case of Sinclair et al. (1998), no corrections for clustering or multiple comparisons were needed. 

8. The WWC-computed average effect size is a simple average rounded to two decimal places. The average improvement index is calculated from the average effect size. 
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Appendix A3.3 Summary of study findings inciuded in the rating for the compieting schooi domain^ 









Author’s findings from the study 














Mean outcome 
(Standard deviation 




WWC calculations 




Outcome measure 


Study 

sample 


Sample size 
(schools/ 
students) 


Check & Connect 

group Control group 


Mean difference^ 
Check & Connect 
- control 


Statistical 
significance^ 
Effect size'' (at a = 0.05) 


Improvement 

index^ 



Sinclair et al., 2005 (randomized controlled trial with attrition problems)^ 


Completed high school or Grade 12 

GED “on time” (%) 


144 30 29 1 

(46) (46) 


0.03 


ns 


1 


Domain average® for completing school 




0.03 


ns 


1 



ns = not statistically significant 

1 . This appendix reports findings considered for the effectiveness rating and the average improvement index. Sinclair et al. (2005) also examined additional outcomes that do not fall within the 
three domains addressed by the WWC review — staying in school, progressing in school, and completing school. These additional outcomes included the likelihood that the student would 
transfer to another school and the level of the student’s participation in the special education transition program. These additional results are not summarized in this report. 

2. The standard deviation across all students in each group shows how dispersed the participants’ outcomes are: a smaller standard deviation on a given measure would indicate that participants 

had more similar outcomes. For binary (“zero-one”) outcomes, standard deviations were calculated using the following formula, ^n/(n -i)*7p*0 “pJ ’ P percentage of the sample 

with a value of 1 for the outcome and N is the sample size. 

3. Positive effect sizes represent effects in the desired direction; negative effect sizes represent effects in the undesired direction. 

4. For an explanation of the effect size calculation, see Technical Details of WWC-Conducted Computations . 

5. Statistical significance is the probability that the difference between groups is a result of chance rather than a real difference between the groups. 

6. The improvement index represents the difference between the percentile rank of the average student in the intervention condition and that of the average student in the comparison condition. 
The improvement index can take on values between -50 and +50, with positive numbers denoting favorable results. 

7. The level of statistical significance was reported by the study authors or, where necessary, calculated by the WWC to correct for clustering within classrooms or schools and for multiple 
comparisons. For an explanation about the clustering correction, see the WWC Tutorial on Mismatch . See Technical Details of WWC-Conducted Computations for the formulas the WWC used to 
calculate statistical significance. In the case of Sinclair et al. (2005), no corrections for clustering or multiple comparisons were needed. 

8. The WWC-computed average effect size is a simple average rounded to two decimal places. The average improvement index is calculated from the average effect size. 
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Appendix A4.1 Check & Connect rating for the staying in schooi domain 



The WWC rates interventions as positive, potentiaiiy positive, mixed, no discernibie effects, potentiaiiy negative, or negativeJ 

For the outcome domain of staying in schooi, the WWC rated Check & Connect as having positive effects. The remaining ratings (potentiaiiy positive effects, mixed 
effects, no discernible effects, potentiaiiy negative effects, and negative effects) were not considered because Check & Connect was assigned the highest applicabie 
rating. 



Rating received 

Positive effects: Strong evidence of a positive effect with no overriding contrary evidence. 

• Criterion 1 : Two or more studies showing statistically significant positive effects, at least one of which met WWC evidence standards for a strong design. 

Met. Check & Connect had two studies meeting WWC evidence standards, one of which met WWC evidence standards for a strong design. Both 
studies reported statisticaiiy significant positive effects on the staying in schooi domain. 

• Criterion 2: No studies showing statisticaiiy significant or substantially important negative effects. 

Met. The WWC analysis found no statistically significant or substantively important negative effects on this domain. 



1 . For rating purposes, the WWC considers the statistical significance of individual outcomes and the domain level effect. The WWC also considers the size of the domain level effect for ratings of 
potentially positive or potentially negative effects. See the WWC Intervention Rating Scheme for a complete description. 



(continued) 
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Appendix A4.2 Check & Connect rating for the progressing in schooi domain 



The WWC rates interventions as positive, potentiaiiy positive, mixed, no discernibie effects, potentiaiiy negative, or negativeJ 

For the outcome domain of progressing in school, the WWC rated Check & Connect as having potentially positive effects. It did not meet the criteria for positive 
effects, because it only had one study that examined outcomes in this domain. The remaining ratings (mixed effects, no discernible effects, potentially negative effects, 
negative effects) were not considered because Check & Connect was assigned a higher rating. 

Rating received 

Potentially positive effects: Evidence of a positive effect with no overriding contrary evidence. 

• Criterion 1 : At least one study showing a statistically significant or substantively important positive effect. 

Met. Check & Connect had one study that showed statistically significant effects on the progressing in school domain. 

• Criterion 2: No studies showing a statistically significant or substantively important negative effect. Fewer or the same number of studies showing indeterminate 
effects than showing statistically significant or substantively important positive effects. 

Met. The WWC analysis found no indeterminate, statistically significant negative, or substantively important negative effects in this domain. 

Other ratings considered 

Positive effects: Strong evidence of a positive effect with no overriding contrary evidence. 

• Criterion 1 : Two or more studies showing statistically significant positive effects, at least one of which met WWC evidence standards for a strong design. 

Not met. Check & Connect had only one study meeting WWC evidence standards that reported on the progressing in school domain. 

• Criterion 2: No studies showing statistically significant or substantially important negative effects. 

Met. The WWC analysis found no statistically significant or substantively important negative effects on this domain. 



1 . For rating purposes, the WWC considers the statistical significance of individual outcomes and the domain level effect. The WWC also considers the size of the domain level effect for ratings of 
potentially positive or potentially negative effects. See the WWC Intervention Rating Scheme for a complete description. 



(continued) 
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Appendix A4.3 Check & Connect rating for the compieting schooi domain 



The WWC rates interventions as positive, potentiaiiy positive, mixed, no discernibie effects, potentiaiiy negative, or negativeJ 

For the outcome domain of compieting school, the WWC rated Check & Connect as having indeterminate effects. It did not meet the criteria for positive effects, 
potentially positive effects, mixed effects, potentiaiiy negative effects, or negative effects because no statistically significant or substantively important findings either 
positive and negative were reported on this domain. 

Rating received 

No discernible effects: No affirmative evidence of effects. 

• Criterion 1 : None of the studies shows a statisticaliy significant or substantiveiy important effect, either positive or negative. 

Met. In the one Check & Connect study that reported on the completing school domain, the effect was neither statistically significant nor substan- 
tively important. 

Other ratings considered 

Positive effects: Strong evidence of a positive effect with no overriding contrary evidence. 

• Criterion 1 : Two or more studies showing statistically significant positive effects, at least one of which met WWC evidence standards for a strong design. 

Not met. Check & Connect had only one study meeting WWC evidence standards that reported on the completing school domain. 

• Criterion 2: No studies showing statistically significant or substantively important negative effects. 

Met. The WWC analysis found no statistically significant or substantively important negative effects in this domain. 

Potentially positive effects: Evidence of a positive effect with no overriding contrary evidence. 

• Criterion 1 : At least one study showing a statistically significant or substantively important positive effect. 

Not met. The WWC analysis found no statistically significant or substantively important positive effects in this domain. 

• Criterion 2: No studies showing a statistically significant or substantively important negative effect. Fewer or the same number of studies showing indeterminate 
effects than showing statistically significant or substantively important positive effects. 

Not met. All findings in this domain were indeterminate. The WWC analysis found no statistically significant or substantively important effects in 
this domain. 

Mixed effects: Evidence of inconsistent effects as demonstrated through EITHER of the following. 

• Criterion 1 : At least one study showing a statistically significant or substantively important positive effect. At least one study showing a statistically significant or 
substantively important negative effect, but no more such studies than the number showing a statistically significant or substantively important positive effect. 

Not met. The WWC analysis found no statistically significant or substantively important effects in this domain. 

OR 



1 . For rating purposes, the WWC considers the statistical significance of individual outcomes and the domain level effect. The WWC also considers the size of the domain level effect for ratings of 
potentially positive or potentially negative effects. See the WWC Intervention Rating Scheme for a complete description. 
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Appendix A4.3 Check & Connect rating for the compieting schooi domain (continued) 



Criterion 2: At least one study showing a statistically significant or substantively important effect, and more studies showing an indeterminate effect than showing 
a statistically significant or substantively important effect. 

Met. The WWC analysis found no statistically significant or substantively Important effects in this domain. 

Potentially negative effects: Evidence of a negative effect with no overriding contrary evidence. 

• Criterion 1 : At least one study showing a statistically significant or substantively Important negative effect. 

Not met. The WWC analysis found no statistically significant or substantively Important negative effects in this domain. 

• Criterion 2: No studies showing a statistically significant or substantively important positive effect, OR more studies showing statistically significant or substan- 
tively important negative effects than showing statistically significant or substantively important postive effects. 

Met. The WWC analysis found no studies with statistically significant or substantively important positive effects in this domain. 

Negative effects: Strong evidence of a negative effect with no overriding contrary evidence. 

> Criterion 1 : Two or more studies showing statistically significant negative effects, at least one of which met WWC evidence standards for a strong design. 

Not met. The WWC analysis found no statistically significant negative effects in this domain. 

• Criterion 2: No studies showing statistically significant or substantively Important positive effects. 

Met. All findings in this domain were indeterminate. The WWC found no statistically significant or substantively important positive effects in this 
domain. 
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